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Abstract Body 


Background / Context: 

Under the No Child Left Behind Act of 2002 (NCLB), third party vendors contract with districts 
and receive public, Title I money to provide out of school time (OST) tutoring to low income 
students. Since 2009, our research team has partnered with six urban school districts to evaluate 
the implementation and impact of these federally-funded OST programs on student reading and 
math achievement. In this paper, we specifically focus on mapping the nature of digital tutoring 
and examining its impact on student outcomes. 

There is considerable variation in how digital OST tutoring providers describe what they do, the 
actual services they offer, how students access these services, and what is delivered. Yet, there is 
little evidence to date that digital providers are increasing the quality of OST tutoring or helping 
historically disadvantaged students to gain ground academically. A report to the WT Grant 
Foundation by Pam Stevens (2012) recently conveyed that end-users of afterschool research have 
found the research output in this area lacking in rigor compared to other bodies of study on youth 
interventions, and the dearth of reliable research that specifically examines digital tutoring 
effectiveness in the OST context is particularly striking. Therefore, our research draws upon the 
emerging research base on quality digital instruction (e.g. Cavanaugh, 2004; Means, 2010; Zhao 
et al, 2005) and digital schooling in the context of underserved students (e.g. Goslee & Conte, 
1998), as well as the considerable research base identifying factors that contribute to high-quality 
OST tutoring in general (Beckett et al., 2009; Elbaum et al., 2000; Farkas & Durham, 2006; 
Lauer et al., 2006; Little et al., 2008; Vandell et al., 2007). 

Purpose / Objective / Research Question / Focus of Study: 

This paper will build on our mixed method longitudinal study of OST tutoring to bring 
intensified focus to the use of online or digital forms of tutoring. To this end, we structure our 
examination around following core research questions: 

1 . What are the key characteristics of different program models in digital OST tutoring, 
and how do they compare or contrast with tutoring implemented without digital tools? 

2. What are the impacts of OST tutoring interventions on student reading and 
mathematics achievement, and what attributes of digital and non-digital programs (and 
their implementation by providers and management by districts) drive the observed 
effects? 

A “digital” tutoring provider is one that uses a digital platform (software or live tutor via a 
technological platform, such as a computer, netbook, or handheld device) as an intentional, 
integral, and consistent part of its instructional delivery strategy. Digital tutoring has become an 
important component of the private tutoring industry, driven in large part by advances in 
technology that allow it to compete with features of face-to-face tutoring and offer the promise 
of scale and low cost implementation. Indeed, digital providers are gaining market share in OST 
tutoring programs at a faster rate than other providers. In one of our study sites, online tutoring 
companies reached a student “market share” as high as 88 percent in one year, and in another 
district, we have observed a single digital provider delivering tutoring to more than 14,000 
students. NCLB mandated unfettered choice in tutoring providers and accordingly gave 
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providers the flexibility to try varied formats for tutoring. However, the implementation and 
effects of the wide range of approaches and fonnats that are emerging in digital tutoring are 
especially difficult for school districts to monitor and assess. We will draw upon our working 
index for categorizing models of digital tutoring, linking the rich descriptive data on specific 
digital program characteristics to a range of program outcomes, such as student engagement, 
content coverage and mastery, and student retention. We will then quantify these data and use 
them in (large sample size) empirical analyses of digital tutoring effects on student achievement. 

Setting: 

Both quantitative and qualitative research settings include six urban school districts: Chicago, 
Milwaukee and Minneapolis Public Schools, Austin and Dallas Independent School District and 
Los Angeles Unified School District. 

Population / Participants / Subjects: 

Our quantitative analysis examines student outcomes and participation trends of eligible and 
enrolled students, with an estimated total sample size (across districts) of 570,000 students. 
Beyond the baseline criteria of students’ eligibility for free and reduced lunch, districts detennine 
additional eligibility criteria, which often includes proficiency levels on standardized 
assessments. There are 180 providers in the quantitative sample, approximately a quarter of 
which are “digital providers”. Our qualitative sample of five digital providers represents 
subcategories of digital providers, including: synchronous, asynchronous, all digital and blended. 
The qualitative sample also includes both local and national providers operating in our sample 
districts in our study, and in some cases, many other districts across the country. 

Intervention / Program / Practice: 

School districts across the country are spending millions of Title I dollars on tutoring for 
economically and academically disadvantaged students. Currently under NCLB, public schools 
that do not make adequate yearly progress for three consecutive years are required to offer 
children in low-income families the opportunity to receive extra academic assistance through 
free, out of school time (OST) tutoring, or “Supplemental Educational Services” (SES). As of the 
2012-13 school year, newly-granted federal waivers are allowing two-thirds of the states to opt 
out of core tenets of NCLB, including SES. Many school districts are continuing to offer 
extended learning opportunities in the form of OST tutoring interventions to students most in 
need and/or to launch new or modified interventions. Digital tutoring programs are playing a 
major role in both the original SES context, as well as post-waiver, district-structured OST 
programs. 

Research Design: 

Our longitudinal, mixed-method design integrates rigorous, quasi-experimental analysis of OST 
tutoring program impacts on student achievement with an in-depth, comprehensive examination 
of the intervention — provider instructional practice in different program models and settings, the 
nature and quality of tutoring provided and district-level program administration — in and across 
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six large, urban school districts. Our analytic process is part of our fully integrated mixed- 
method research design where the quantitative and qualitative teams coordinate and collaborate 
at all stages of the study (design, collection, analysis, and dissemination). Specifically, the 
qualitative and quantitative teams meet to review analytical findings from both study 
components, direct additional data collection, refine analysis plans, and prepare dissemination of 
findings to stakeholders. 

Data Collection and Analysis: 

Our analytic process is part of our fully integrated mixed-method research design where the 
quantitative and qualitative teams coordinate and collaborate at all stages of the study (design, 
collection, analysis, and dissemination). Specifically, the qualitative and quantitative teams meet 
monthly to review analytical findings from both study components, direct additional data 
collection, refine analysis plans, and prepare dissemination of findings to stakeholders. 

The qualitative data are drawn from observations of full tutoring sessions using a standardized 
instrument developed to capture OST tutoring in practice, including those elements of instruction 
specific to the digital context. In addition to observations, qualitative data sources include 
personal interviews with district staff, provider administrators and tutoring staff, focus groups 
with parents of eligible students, and curriculum analysis. We developed and use common 
protocols, by role group, and in our observation of tutoring sessions. We use a constant 
comparative method (both within and across method) to develop and refine our understanding of 
patterns and dissimilarities in tutoring practices across providers. 

Quantitative analysis relies on three econometric strategies in estimating digital and non-digital 
OST tutoring effects (to address research question #2) include: 1) value-added modeling 
(following Zimmer et ah, 2007 and Heinrich & Nisar, forthcoming), 2) fixed effects models 
(student and school fixed effects), and 3) generalized propensity score matching methods. Our 
two comparison groups will consist of students eligible for OST tutoring in each district who: (1) 
do not receive tutoring, or (2) receive non-digital tutoring services. Our application of multiple 
quasi-experimental strategies enables us to test the sensitivity of our results to alternative 
assumptions about student selection into tutoring programs and their selection into different 
types of providers (e.g., digital or non-digital, and variants of digital tutoring), which are also 
informed (in specification) by the qualitative research. In addition, administrative data from the 
school districts’ and providers’ implementation of tutoring interventions will allow for the 
construction of measures of student tutoring dosages with specific providers, including number 
of hours attended and total expenditures. 

Findings / Results: 

Digital providers are gaining market share at a faster rate than providers of face-to-face private 
tutoring. Further, online providers tend to charge significantly more per hour ($20 more per hour, 
on average) and provide students with fewer hours of service than face-to-face tutoring providers 
(18 hours versus 35 hours). These higher rates could be justified if students and families were 
getting higher quality services for their money. However, estimates of the comparative 
effectiveness of digital vs. non-digital providers in our study do not appear to justify their 
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differential rates. Indeed, by our own estimates, digital tutoring is, on average, negatively 
correlated with student performance in mathematics and reading (i.e., when comparing students 
served by digital vs. non-digital providers) (Burch et al, 2011). Coupled with on our in depth 
qualitative research into the nature of the instructional setting in digital OST, the factors 
impacting the design of the instructional setting, and its impact on student learning, we identify 
several characteristics or conditions that we hypothesize are likely to detennine digital tutoring 
effectiveness: access (including hardware and software), the role of the tutor, and the nature of 
curriculum and assessment. Ultimately, we are capturing how decisions made about hardware, 
software, the role of the tutor, curriculum and assessment influence instructional quality and 
ultimately, the impact (or lack thereof) on student learning. 

Conclusions: 

Because digital tutoring is rapidly expanding, more rigorous, independent evaluations of their 
effectiveness is critical to inform federal, state and local policy decisions that influence their role 
and application of technology in educating underserved students. Further, the in-depth 
observations and vignettes in this paper illustrate the challenges in documenting and measuring 
technology use and its impact in OST tutoring interventions on student learning. Currently, the 
limited, self-generated information that is disseminated by providers to parents and students does 
not usefully guide parent and student choices of OST tutoring providers. In summary, the paper 
will generate important and generalizable insights on the nature of digital OST instruction, what 
constitutes and contributes to high-quality, OST tutoring in digital contexts. 
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